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ABSTRACT 

This paper assesses the value of research, not on its 
methodology type but on its goal and accomplishments. It outlines an 
idea for developing a knowledge base capable of informing teacher 
education program planning and other educational policy decisions. 
Studies are described that advance knowledge about causal 
relationships between classroom processes and student outcomes, 
especially studies that document and describe desirable practices. 

The paper is divided into the following sections: (1) "Introduction”; 

(2) "Factors that Contribute to the Value of Research Studies”; (3) 
"Quantitative Methods"; (4) "Qualitative Methods"; (5) "Shift in 
Focus from Methods to Questions"; and (6) "Needed Research in Social 
Studies." (EH) 



*:Vn:r: v:sr 



?c >f tt ' 



* it * i: * it * it ' 



'? V: it it it it it : 



■ it * * * it it it it it V: it it it it it it it it it it i: it it it it it it it it it i 



Reproductions supplied by EDRS are the best that can be made 
from the original document, 



it it ?? v* 



ft it it it it it it it it it it it it it it it it it it it it it it < 






c i: it it it it it it it it it it it it it it it it it it it it it 



ED 392 734 



THOUGHTS ON THE QUALITATIVE -QUANTITATIVE DEBATE 
Jere Brophy 

Michigan State University 



\\ 



V 



/ 

J 



PERMISSION TO REPRODUCE THI< 
AS BEE N CRANTED 





t O THE EDUCATIONAL resources 
INFORMATION CEN TER <ERiCl 



educational resources information 
/ CENTER, ERIC* 

LVOt ' ’ eD,0d ‘- CP <T 

° ,v ‘ K Person or organ,/*? r „, 

□ V ”n ■ \u* rv*c,> m.,do to 

»«»p- 






pn »'K o! \ o.\ • i 

J c invsj jj.. •», 

H « • i J OFMi i- 



’’ suiod >•> thK 

>! *’o< css.in!v 
v!r-r- • of i. ,- N 



This presentation was made as part of a symposium entitled "The 
Quantitative-Qualitative Debate and its Relevance for Social Studies Education: 
Still Alive and Kicking or Beating a Dead Horse?" held at the annual meeting of 
the College and University Faculty Assembly of the National Council for the 
Social Studies, Chicago, November, 1995. 
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My views on research methodology are rooted in the fact that although I 
take an interest in developing grounded theory, I approach research on teaching 
primarily from a practical rather than a theoretical standpoint. I prefer to 
work primarily inductively, by first establishing empirical information about 
variables and their relationships, then identifying the causal linkages that 
explain these relationships and developing implications for teachers. 

I view both teaching and research as means, not ends. Teaching is action 
taken to help students make progress toward instructional goals. Research is 
action taken to answer some question. To evaluate either teaching or research, 
we need to consider both the importance of their goals and the degree to which 
they get accomplished. Good teaching is good because it succeeds in helping 
students attain worthwhile learning goals. Good research is good because it 
succeeds in answering worthwhile questions. 

From this perspective, attempts to argue qualitative versus quantitative 
research methods make no more sense than arguing the value of hammers versus 
screwdrivers. To argue the relative merits of tools productively, we need to 
consider the purposes and goals for which they might be used. Tools offer 
trade-offs. They are ideally suited for some purposes, usable but not ideal 
for other purposes, and irrelevant or counterproductive for still other pur- 
poses. This is as true for research methods as it is for other tools. Any 
systematic, clearly described, replicable method is scientifically acceptable, 
but particular methods are more suited to some purposes than others. 

Sensible research planning begins with goals. Presumably, the research 
is needed because some information is needed- -the test of the hypothesis, the 
answer to a question, or some other knowledge to be generated. Clarifying 
research goals allow us to identify the kinds of data that we need to achieve 
our goal. In turn, achieving clarity about needed data allows us to identify 
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the research methods needed to produce those data. It is only at this stage of 
research planning, after clarification of research goals and data needs, that 
we should turn to methodological issues. We then can consider relevant methods 
with an eye toward their potential for generating the needed data, their cost 
effectiveness, and other criteria. 

I believe that scientists who proceed in this goals - or iented fashion are 
the most likely to investigate important questions using methods that are well 
suited to those questions. In the process, they may need to invent new meth- 
ods, if the needed information cannot be generated using existing methods. 

These invented methods are then available for use by others, who may apply 
them, for good or ill, in ways that were not foreseen by their inventors. Sci- 
entific advances, especially in young sciences such as research on teaching, 
often are stimulated at least as much by tool invention as by theoretical de- 
velopments . 

However, researchers sometimes fall in love with their methods. Some of 
them keep doing the same kinds of studies, even when there no longer are good 
reasons to do so. This leads to work that is methodologically elegant but sub- 
stantively trite. Others shift to new questions but cling to their old meth- 
ods, even when these are not well suited to the questions ostensibly under in- 
vestigation. People whose methodological commitments are this strong and mis- 
directed are no longer acting primarily as scientists seeking to generate 
needed knowledge. Instead, they are behaving compulsively for overdetermined 
reasons, treating a means as if it were an end in itself. 

When I assess the value of research, I focus not on its methodological 
type but on its goals and accomplishments. Was it designed to generate knowl- 
edge that would contribute significantly to the literature and carry important 
theoretical or practical implications? If so, I will be eager to find out 
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more, both about the methods the authors used to address their questions and 
the answers they developed. If I don't see much need or reason for the study, 
however, I am unlikely to pay much attention to it or cite it in my future 
writings, even if it was elegantly implemented. For example, I would value 
highly a quantitative study that contributed new information about, the trade- 
offs involved in using different instructional methods, as well as a qualita- 
tive study that offered thick description of the instructional beliefs and be- 
haviors of a teacher who had been identified as outstanding using criteria that 
I found compelling. I would have much less interest in a quantitative study 
that merely replicated already well -understood relationships or a qualitative 
study that offered thick description of the beliefs and behavior of a teacher 
of unknown effectiveness who had been selected for study for no particularly 
good reason. 

Because of my personal interests in developing a knowledge base capable 
of informing teacher education program planning and other educational policy 
decisions, I am most interested in studies that advance our knowledge about 
causal relationships between classroom processes and student outcomes, espe- 
cially studies that document and describe desirable practices. I am less in- 
terested in studies that call attention to problems but do not suggest solu- 
tions to those problems, as well as studies that generate information that 
might be nice to know buc is lacking in clear implications for improving educa- 
tion (e.g., statistical trends in various educational indicators, group differ- 
ences in achievement or attitudes, or developmental trends in children's knowl- 
edge presented without consideration of how to stimulate advances through in- 
struction) . 

Other researchers attach different priorities to the kinds of knowledge 
that is generated by different types of research. Clearly, value-based 
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decision making is involved in determining what is needed information that 
would constitute a significant contribution to the literature. More could be 
said about the relative value of different educational research goals, but at 
this point I want to shift from goals to methods. Therefore, most of the 
subsequent discussion will assume that the researcher's goals are worthy ones, 
and will focus on methodological issues. 

Factors That Contribute to the Value of Research Studies 
I believe that three main factors contribute to the value of a research 
study as a significant contribution to the literature. Two of these are im- 
plied in what was said previously and will be mentioned only briefly here. 

First, there should be a need for the information that the study is designed to 
generate. We do not need further studies, qualitative or quantitative, to dem- 
onstrate that behavior can be shaped through reinforcement or that classroom 
discourse typically occurs in segments consisting of a teacher question, a stu- 
dent response, and a teacher reaction to that response. 

Second, the research methods used should be appropriate -- capable of gen- 
erating the kinds of data needed to address the question ostensibly under in- 
vestigation. If the question involves establishing cause -and- effect relation- 
ships, experimental methods will be needed. If the question involves correla- 
tions or other relationships between variables, some form of quantification and 
a sample large enough to allow for statistical testing will be needed. If the 
question involves developing deep understandings of how particular practitio- 
ners (teacher educators, principals, teachers) approach particular aspects of 
their jobs, thick description methods will be needed. 

Most of the important questions about good practice will require ambi- 
tious programs of research that involve multiple studies using these methods 
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and many others. Typically, a knowledge base develops primarily through repli- 
cation of key findings by different investigators using different methods 
rather than through some quantum effect of a single elegantly designed study. 
Even after a finding becomes well established as a principle or generalization, 
research is needed to establish the parameters of its applications and to il- 
lustrate how it plays out in particular situations. 

We need to appreciate both the strengths and the limitations of various 
research methods. Elegant correlational studies provide useful information 
about associations between variables but not about cause -and- effect relation- 
ships; elegant case studies of outstanding practitioners provide useful infor- 
mation about what is possi K le to accomplish but not about the percentage of 
practitioners who might reach this level and what it might take to get them 
there. If investigators are determined to answer particular questions, they 
will need to use research methods capable of producing data that sp^ak to those 
questions. If they fail to do so, they may produce useful information of vari- 
ous sorts, but not information that speaks directly to the question they wished 
to address. 

Besides considering the need for the information to' be generated and the 
fit between the ostensible research question and the methods used in the study, 
I believe that a study's value as a contribution to the literature is deter- 
mined by the sophistication with which the methods are implemented. The meth- 
ods may be as diverse as survey research, systematic classroom observation, 
stimulated recall interviewing, or qualitative analysis of instructional 
materials. These and any other research methods may be implemented on a 
continuum from very poorly to very well. To illustrate, I will focus on two 
methods that people frequently have in mind when they talk about quantitative 
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and qualitative approaches to research: classroom interaction coding systems 

and thick description methods for studying classroom processes. 

In speaking of sophistication factors, I do not refer to the technologi- 
cal features of the data collection instruments or the complexities of the sta- 
tistics used to analyze the data. Instead, I have in mind the degree to which 
the investigators bring to bear sound research logic and knowledge of the set- 
ting in which the research will be conducted so as to make good decisions about 
basic sampling, instrument design, and data collection and analysis. Many of 
these sophistication factors involve the craftsmanship or engineering aspects 
of being a researcher: developing and perfecting the tools that one uses to do 

the research. 

Well-clarified research questions imply certain sampling requirements 
that must be met if the questions are to be addressed effectively. For start- 
ers, investigators who intend to generalize from their findings need to recruit 
samples that are representative of the populations to which they wish to gener- 
alize. If the study is intended to focus on effective teaching, or to contrast 
effective teaching with less effective teaching, it will be important to begin 
with clear criteria of teaching effectiveness and to use these in recruiting 
the sample. 

If the study focuses on curriculum and instruction, it may involve hidden 
assumptions about teacher preparation and classroom management. For example, 
the logic of the study might imply that the teachers are knowledgeable about 
the subject and experienced in teaching it, and also that they are professional 
enough as individuals and skillful enough as classroom managers to enable them 
to elicit good cooperation and attention from their students. If the teacher 
is not the sort of individual that the logic of the study demands, or if what 
occurs during the periods of observation does not correspond well with what was 
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supposed to occur, the recorded information will not be valid as data that 
speak to the questions ostensibly being addressed. Regardless of whether the 
data are quantitative or qualitative, the investigators will end up with a 
story about how teachers' unfamiliarity with the content or inability to 
sustain student engagement negated the intended curriculum and instruction, 
rather than a story about the effects of the intended curriculum and instruc- 
tion on students' learning. 

Clear research goals also carry implications about the kinds of data to 
be collected and the contexts or settings in which relevant classroom processes 
occur. To minimize noise in the system, it will be important to focus data 
collection and analysis on these relevant aspects of the classroom day or week. 
If the study deals with mathematics teaching and learning, collect data during 
math periods; if the study deals with both mathematics -nd science, collect 
data during both math and science periods but record and analyze them sepa- 
rately; and so on. Investigators doing process -product or other quantitative 
studies have sometimes introduced ambiguities into their data by failing to 
confine their observations of teachers to the most relevant and comparable 
times and contexts. Investigators doing thick description case studies have 
introduced ambiguities into their data by trying to record and analyze every- 
thing instead of focusing on data relevant to the study's goals. 

Quantitative Methods 

The best quantitative studies avoid these problems and also display other 
sophistication factors: Data collection, aggregation, and analysis methods 

allow for separate analyses of data from different contexts (one can always 
combine across contexts later if the separation doesn't yield anything signifi- 
cant); to facilitate comparisons across teacners, scores can be expressed not 
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only as raw frequencies but as relative frequencies or percentages (e.g., per- 
centage of good answers that were praised, percentage of lessons begun with an 
overview); patterns of initiation and reaction or other sequential information 
can be retained in the coding; coding systems can be designed to include dis- 
tinctions in the quality, timing, or appropriateness of teacher behaviors along 
with recording of their occurrence; different coding systems can be tailored 
for different subject matters or types of activities; classrooms need to be ob- 
served frequently enough to ensure reliability and validity of measurement 
data; reporting should include basic descriptive statistics, not just correla- 
tions or multivariate data; base rates should be kept in mind when interpreting 
correlational relationships (e.g., if the data show a positive correlation 
between teacher criticism of poor work and student achievement gain, but the 
base rate data indicate that such criticism was rare, it would be misleading to 
interpret the correlational finding as indicating that "teachers who frequently 
criticized poor work elicited greater achievement gains from their students.") 
It is also important for quantitative researchers to report findings for each 
variable considered individually rather than only for cluster or combination 
scores. The latter approach masks potentially useful information and may even 
negate the value of the study (as has happened when cluster scores included 
subsets of classroom process variables that correlated in opposite directions 
with student achievement gain) . 

Qualitative Methods 

Concerning qualitative research, leaders in educational ethnography and 
thick description methodology have developed important guidelines for using 
triangulation and other methods for ensuring the reliability and validity of 
descriptive data. I would add the following, based on my analysis of the 
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reasons why I have or have not found various qualitative studies useful. Most 
of these reflect problems with the goals or logic of the study rather than with 
the scientific acceptability of the data. 

First, qualitative studies ought to be planned to develop information 
that educators will find needed or useful. Ethnography for its own sake 
usually does not do this, but thick descriptions of practices that are rich in 
educational policy or teacher education program implications are very valuable. 
These usually are developed when the investigators ask educational rather than 
anthrooo logical questions. It also helps if the study is planned and its 
findings are interpreted with awareness of the full range of educational 
research, not just educational ethnography. 

Second, don't lose the forest for the trees. Focus on data of interest 
to the question under investigation, in the original data collection, in subse 
quent analyses, and in reporting the findings. 

Third, incorporate relevant quantitative data (e.g., if the study deals 
with differential teacher response to male versus female students during public 
discourse, the report might include tables showing the numbers of different 
types of questions directed to boys versus girls, the percentages of their re- 
sponses that were praised and criticized, and so on) . Thick description data 
also offer opportunities for microanalyses of the timing of events and other 
forms of data presentation that are not easily classified as either quantita- 
tive or qualitative. For example, Alton-Lee, Nuthall, and Patrick (1993) used 
fine-grained analyses of the timing of classroom events to support their resis- 
tance theory interpretations of minority students' disruptive behavior. They 
showed that such behavior tended to occur immediately following events that un- 
derscored the Eurocentric nature of the historical content being discussed in 
the class. 
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Fourth, I find more value in reports that not only describe what the 
teacher does, h't critique it- -noting what is done well or poorly, and with 
vh/’t effects on Ss*.. dents. Such reports document the teacher's thinking but do 
not automatically assume its validity. They treat teachers as informed profes- 
sionals whose views are worthy of respectful attention, but also recognize that 
some of these views are misinformed, defensively distorted, or otherwise 
invalid. 

Fifth, I find qualitative reports more valuable when they consider stu- 
dent effects on teachers as well as teacher effects on students, especially 
when focusing on gender, race, or social class issues. Many qualitative inves- 
tigators, especially those heavily into critical theory notions, depict teach- 
ers as proactive and powerful socializers of students into an oppressive sys- 
tem. This ignores a great deal of other research suggesting that teachers not 
only are much less powerful than that but often act out of fear of losing 
control of the class, and that much of what they do is better interpreted as 
responsive behavior forced or at least elicited by student initiated actions 
than as proactively initiated teacher effects on students. 

Sixth, I find qualitative reports more valuable when, along with describ- 
ing the specific, they consider the potentially generalizable . What generali- 
zations about life in classrooms or principles of good teaching are suggested 
by the study, at least as hypotheses? Qualitative researchers tend to take 
offense when they hear other researchers respond to their work as “just sto- 
ries," usually because they take this as evidence that their work is being 
rejected as unscientific. My impression, however, is that this comment usually 
expresses, not the perception of a lack of scientific method, but the percep- 
tion of a lack of significant, generalizable findings emerging from the study. 
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In other words, when critics say "It was just a story," what they really mean 
is "It was a rambling story that had no moral." 

Finally, I find qualitative reports more valuable when they point to and 
thickly describe desirable practices rather than merely bemoan practices that 
they view as inappropriate. I would be even more pleased and impressed, of 
course, if the investigators mounted follow-up experimental studies designed to 
test their inferences about how to improve teaching. I think that they have 
some responsibility to do this if their work emphasizes policy advocacy rather 
than mere description of observed practice, and especially if the feasibility 
or advisability of the policies they advocate may be questioned on the basis of 
existing data or wisdom of practice. 

Shift in Focus From Methods to Questions 

Both quantitative and qualitative methods involve interpreting classroom 
processes. Quantitative methods offer the opportunity to use inferential sta- 
tistics, but they run the risk of distorting the meanings of observed events by 
forcing them into categories that they don't really fit, stripping away poten- 
tially important contextual qualifications, or aggregating data across differ- 
ent contexts in ways that create scores that have convenient statistical prop- 
erties but ambiguous meaning and questionable comparability. Qualitative meth- 
ods offer the opportunity to retain the coherence of the story, but they run 
the ’-isk of attributing events to personal rather than situational causes, im- 
posing the observer's agenda in recording and interpreting events, and develop- 
ing conclusions based on what may turn out to be an unrepresentative case. In 
developing ways to guard against or compensate for these risks, quantitative 
investigators have begun to take better account of context and to develop ways 
to retain some of the continuity of the classroom interaction they record. 
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Meanwhile, qualitative researchers have begun to do more purposeful sampling, 
to use triangulation and other methods of grounding and testing their 
interpretations, and to incorporate numerical data representations into their 
analyses . 

When done well, both methods reveal thoughtful efforts to record and 
interpret data in ways that reflect an educator's perspective in separating 
figure from ground, determining what is central versus peripheral, and identi- 
fying the probable cause -and- effect mechanisms that might explain the 
relationships observed. The more sophisticated educational research being 
conducted currently uses multiple methods and takes into account a broad range 
of factors including the purposes and goals of the instruction, curriculum 
content and materials, lesson and activity formats, teacher and student 
behavior and interactions, the classroom climate and other social and affective 
factors, and anything else that might be relevant to the questions addressed. 
The resulting richness and diversity of data go well beyond what is usually 
implied in discussions of quantitative and qualitative methods. As an example, 
see the study by Turner (1995) on situated motivation in literacy instruction, 
which won the International Reading Association's outstanding dissertation 
award for 1992-93. 

Over the past 15 or 20 years, the qualitative versus quantitative debate 
has receded in intensity and become transformed into more productive discus- 
sions. One reason for this has been increasing recognition that methods cannot 
be debated independently of the purposes to which they are to be applied, so 
that the formulation "qualitative versus quantitative methods" is an oversim- 
plified misframing of the question. A second and more important reason why 
this debate has receded is that we have moved past it. By "we," I mean those 
of us who came into education from another discipline, bringing our disciplin- 



ary purviews and tools with us. Those of us who came in from psychology or 
sociology tended to bring a primarily quantitative tools; those who came in 
from anthropology tended to bring primarily qualitative tools. Some research- 
ers from these disciplines have collected data in classrooms but remained 
primarily psychologists, sociologists, or anthropologists. Others, however 
(and I include myself in this group), have gradually made a transition, shift- 
ing from their former discipline to education as their primary identification. 

In the process, they have begun adopting educators' purviews and addressing 
questions that are primarily educators' questions, not social scientists' 
questions . 

As a result, education has developed and coalesced as a domain of schol- 
arship and research in its own right. It is informed by the social sciences 
but has begun to accumulate its own knowledge base and research paradigms, 
comparable to medicine with respect to the biological sciences or engineering 
with respect to the physical sciences. This is resulting in the generation of 
much more knowledge that has validity for and is directly relevant to the con- 
cerns of educators. As these developments progress, we can look forward to 
increased emphasis on studying significant educational questions using whatever 
methods are best suited for addressing these questions (usually these will be 
multiple methods), and less research that is easily classified as quantitative 
or qualitative or as educational psychology, sociology, or anthropology. In my 
view, this is all to the good, and long overdue. 

Needed Research in Social Studies 

Participants in this symposium were asked to identify types of research 
that are particularly needed in social studies. I will respond to this request 
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based on my own priorities, although I encourage any research that informs 
decision making about policy and practice in social studies education. 

Concerning traditional forms of research that subsume old qualitative 
versus quantitative arguments, I believe that social studies education would 
benefit from more process -outcome studies. However, I do not mean the kinds of 
process - outcome studies that were done 20 years ago, when processes were 
measured primarily with frequency counts aggregated across contexts and without 
adequate consideration of instructional goals, and outcomes were restricted to 
standardized tests. Instead, I mean quasi -experimental studies of the trade- 
offs involved in implementing coherent models of curriculum and instruction. 
Process measurement would be informed by thorough knowledge of the curricular 
purposes, content sources, and instructional methods in the model or models 
being studied, and would include thick description of their implementation that 
afforded opportunities for analyzing the coherence of the instruction and the 
degree to which it was congruent with the model, in addition to more 
conventional analysis of process-outcome relationships. The outcome measures 
would include multiple and authentic assessments focusing on the major goals of 
the instruction, including not only knowledge goals but skill, value, and 
citizen participation goals (to the extent that these were included among the 
intended outcomes). The ultimate payoff of this kind of research would be a 
knowledge base composed of thick description of what is involved in 
implementing models known to be effective as vehicles for attaining their 
stated goals, along with information about the trade-offs embodied in selecting 
these models (i.e., cost-benefit considerations, foregone opportunities to 
pursue alternative goals) . 

Various forms of research and scholarship, much of it less traditional 
than the kinds of studies that education has borrowed from psychology, 
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sociology, and anthropology, is needed to inform the development of 
instructional models. Curricular questions cannot be resolved through purely 
empirical methods because they involve value questions. However, curricular 
arguments always contain implied assumptions that can be tested empirically. 
These include readiness assumptions (i.e., that students at a given grade level 
are ready to learn particular content) and transfer assumptions (i.e., that 
mastery of such content will enable the students to handle certain life 
situations effectively). I think it is important for social studies educators 
to clarify and test the empirical claims embedded in the rationales for 
curricular models. 

I have been pursuing this kind of work with respect to curricular models 
that emphasize connecting with students' existing knowledge and experience and 
engaging them in actively constructing new knowledge and correcting 
misconceptions. This approach presupposes the availability of a knowledge base 
describing what children typically know (or think they know) about the content 
taught at their respective grade levels. Curriculum developers and teachers 
then can use this information as the basis for developing instruction that both 
builds on the students' existing valid knowledge and addresses their 
misconceptions . 

In mathematics and science, parallel knowledge bases have been developed 
by interviewing students before and after they experience instructional units. 

I believe that similar work is needed in social studies. In collaboration with 
Bruce VanSledr ight , I have conducted such research at the fifth-grade level on 
students' knowledge and thinking about U.S. history (Brophy & VanSledr ight , 
1996), and in collaboration with Janet Alleman, I am currently collecting 
information about the knowledge and thinking of students in the primary grades 
(K-3) concerning topics typically taught in social studies in those grades 
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(especially cultural universals such as food, clothing, transportation, 
occupations, families, communities, government, etc.). 

In what we call our developmental studies, large samples of students 
stratified by gender, achievement level, grade level, and socioeconomic status 
of their school districts are being interviewed to provide information about 
developments across age and grade in children's knowledge and misconceptions 
about these topics. In what we call our educational studies, students being 
taught experimental units based on instructional plans that we have developed 
are interviewed before and after the units to determine the effects of the 
instruction on their entering knowledge and thinking. 

We think that many more of these educational studies are needed. They 
involve theoretically based determination of significant goals and key ideas 
suitable for teaching at the grade level, review and synthesis of research on 
content vehicles and instructional methods that might be used to teach content 
clusters structured around the key ideas, development of units designed to 
teach these topics from a social education perspective, and cycles of testing 
and revision that include the rich and varied measurement of processes and 
outcomes described previously. Again, the ultimate point is to design coherent 
models of effective social studies instruction, perfect them through testing 
and revision, and in the process collect data that include thick description of 
effective practice and information about the trade-offs involved in opting for 
the model instead of one of its recognized alternatives. 
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